Method, device and arrangement for measurement of app usage based on visual characteristics

ABSTRACT

Method for supplying data to remotely cultivate a digital content reference library, to be executed by an electronic terminal device capable of visually displaying digital content to a user and communicating with a communications network, comprising establishing, by automatic observation logic, visual characteristics data representing the on-screen appearance of digital content, such as application, shown via a display of the terminal device (406), and transmitting the visual characteristics data to a remote server for updating a digital content reference library thereat linking content identity information with associated visual characteristics data in the received or processed form (408). A compatible method to be executed on the network side is presented as well as related, applicable terminal devices and server arrangements.

The invention relates generally to computer applications running ondifferent technological platforms. Particularly, however notexclusively, the present invention pertains to a device, arrangement andmethod for enabling visual characteristics based recognition of thecontext of a user interface and user behavior associated with it, suchas of screen activity, identity of a running application or othercontent.

BACKGROUND

Different media measurement, user profiling, Internet panel, mobilepanels, digital marketing tools and other analytics solutions requireobtaining and analyzing device usage data from a target population andoften also so-called reference groups. The evident goal is to get graspon the underlying trends, habits, problems and needs, whereupon bettersuiting, better functioning, more accurate and also larger audiencereaching products, services and marketing efforts are easier to designamong other potential uses of such data.

Many of the contemporary analytics tools capable of mining dataindicative of user behavior e.g. on mobile or Internet platforms seem toconcentrate on tracking a number of relatively general technicalindicators such as predetermined HTTP (Hypertext Transfer Protocol)events and e.g. active thread/application/window names, oralternatively, exploiting application and/or platform-specific, highlytailored data collection technologies. These approaches, althoughpotentially providing feasible results, limit the versatility of theobtainable data or complicate and slow down the development of theassociated monitoring and analytics tools as they have to beparticularly tuned for each monitored product, respectively. In manycases such solutions further fail to provide validation of actual userbehavior with a particular service, for instance.

Meanwhile, the evolution of media and Internet services such as websites or web-accessible services is faster than ever. Both wired (e.g.desktop computers and smart TVs) and wireless devices (e.g. tablets,phablets, laptops and smartphones) have already changed the way peopleaccess and engage with digital services, and as a result, both thebusiness and technological landscapes are encountering constantturbulence.

Further, user behavior is changing quickly due to parallel use ofmultiple competent devices ranging from the aforesaid smartphones totablets, and e.g. from laptops to smart TVs. Particularly in mobilecontext, the consumers already have a choice from a library of over amillion applications, or ‘apps’, available in the app stores, and theycan opt to use not only native applications but also e.g. HTML5(Hypertext Markup Language 5) applications, widgets, web sites, orsomething in between.

The information resulting from the data collection and mining tools ofthe aforementioned traditional type is simply not adequate anymore inthe light of growing demand for digital analytics and audienceestimations. Instead, it has become important to be able to validatewhat is the app that is being used by the user at any given time, and tobe able separate that from other technical traces of device activity.

A scalable media measurement solution is thus desired. Preferably, itwould be capable of both mobile and Internet measurements and adapt todifferent modern, device-wise often very fragmented, use scenariosbetter than the aforementioned solutions concentrating on monitoring anumber of predefined elements and events on certain specific platformsonly based on e.g. data traffic arising from user-service interactions.

For example, both the markets considering e.g. hedge funds, portfolioanalysts, and investment banks, and mobile/Internet industry playerssuch as device vendors, carriers, app developers, and Internet companiesall alike, would prefer obtaining substantially real-time insight intothe actual use of Internet services—whether used through native apps,HTML5 software, web sites, or e.g. widgets. Holistic projections of thedigital audience, over the long-tail of properties, conducted regularly,e.g. on a daily basis, would undoubtedly be rather useful in supportingfact-based decisions, whether it is about investing into a stock,analyzing competition, or understanding consumer behaviors to gain newusers, minimize churn or to increase user engagement, for instance.Instead of survey data or data on installations, on-device metered reachand engagement data on total and unduplicated usage could indeed beoffered e.g. through a dynamic web reporting dashboard and integratedbusiness intelligence tools, providing near real-time statistics andvarious options for users to extract relevant data cuts and correlationsthemselves, whenever it is needed.

Various events, applications, services, etc. having regard to e.g. userengagement with a terminal device could be monitored and recognizedbased on data indicative of display content and thus also of displayusage in the terminal. Just like a physical fingerprint characterizingthe unique configuration of friction ridges of a human finger, aso-called digital fingerprint could be established from data shown inthe display to describe the associated visual characteristics. Dependingon method selected for establishing the fingerprints, they could beconstructed with uniqueness in mind so that afterwards, based on a merefingerprint the original data, e.g. source application name or activitytype, could be resolved. For example, a fingerprint could be associatedwith a certain application via a mapping table linking applications andfingerprints established therefrom.

Therefore, by comparing a target fingerprint against predefinedreference fingerprints, each of which (references) being also linkedwith desired metadata describing the source data or ‘data source’ to thedesired extent, the target fingerprint could be conveniently associatedwith the correct source data. A matching or best-matchingpre-established fingerprint would be first determined for the targetfingerprint by executing a search/matching algorithm, and then themetadata associated with that (best-)matching pre-established referencefingerprint would be retrieved to uncover the source data to theavailable detail.

Through such visual fingerprinting, a variety of applications andrelated activities could indeed be uniquely identified and their usagetracked for a myriad of purposes. For example, user terminals could sendimage data to a network service that hosts the reference library andperforms the matching and related identification, and potentialsubsequent analysis. However, a further snag remains in the creation ofthe reference (fingerprint) library itself.

One approach to provide the reference library resides in expert basedgeneration, i.e. a plurality of trained experts initially takes ortriggers taking screenshots of different applications or eventsinvolving characteristic display activity and manually link it withsource data such as application identity data, e.g. application name ortype, in the reference library.

Such tedious expert based approach is technically applicable, but it hasits evident drawbacks at least if used as the sole solution for thereference library generation, considering the manual workload required,associated potential human errors, typically still somewhat limitedcoverage and easily rather lengthy update delay between e.g. a launch ofa new application and its actual introduction in the reference libraryin the form of a digital fingerprint and related metadata.

SUMMARY

It is the objective of the invention to at least alleviate one or moredrawbacks related to the prior art.

The objective is achieved by the various embodiments of a method, anarrangement and a device as defined in the appended claims.

In one aspect, a method for cultivating a digital reference library fordigital content, such as a plurality of applications, linking contentidentity information and visual characteristics data, comprises

-   -   receiving content identity information and visual        characteristics data representing and being automatically        created based on the on-screen appearance of the content, such        as application or other content item, in the terminal device by        predetermined observation logic running in the terminal device        for establishing and transmitting at least visual        characteristics data,    -   adapting the digital reference library by updating an entry        therein having regard to said content with the visual        characteristics data in the received or processed form, wherein        the entry is addressable with the content identity information.

In a preferred embodiment, the content identity information is receivedfrom the terminal device that provides the visual characteristics data.Preferably the identity information is indeed automatically establishedeither by the terminal or e.g. a network entity such as a server.Alternatively or additionally, a user of the terminal device could inputthe identity information or at least part thereof via the UI of theterminal optionally responsive to a prompt triggered by the observationlogic. This option could be exploited e.g. in scenarios wherein theautomated logic is incapable of (at least fully) extracting the identityinformation having regard to the screen contentautonomously/automatically.

In one embodiment, the method particularly comprises determining whetherthe reference library includes an entry for the particular content itemin question based on the received identity information, and in the caseof previously non-existing content establishing, and in the case ofalready-existing application, supplementing an entry in the referencelibrary for the indicated application.

In an embodiment, the content identity information indicates at leastone element selected from the group consisting of: applicationidentifier (of the on-screen application), application name of the same,application manufacturer, application type or class, web page address orname, web site address or name, event occurring, activity performed,operating system, and purchase or payment.

In case of a web browser application running in a terminal device, theidentity information and the visual characteristics may relate to thebrowser application itself and/or the content shown therethrough, whichmay refer to a web page, web site or web application, for instance.

The application the visual on-screen characteristics of which arecaptured and provided along with identity information for updating thereference library, may be or comprise a so-called native application.Alternatively it may refer to the aforesaid web application, or‘web-based application’, running through a browser or e.g. an emulatedapplication.

In an embodiment, visual characteristics data representing the displayedon-screen appearance of content (item), such as an application, in aterminal device without application identity information is received andthis new data is matched as is or in processed form with the existingdata in the entries to find the identity of a matching content item. Thevisual characteristics data may be sent by screen or application usagemonitoring logic running in the terminal device.

Accordingly, the identity information associated with the matching entrymay be provided forward. The information may be utilized in thedetermination of content, such as application, usage statistics (e.g.count/frequency), for example, as the visual characteristics dataprovided by the terminal devices reflects the content/applications thatare active and in use at such devices. The derived statistics or otherdeliverables may be exploiting in designing a digital marketing campaignor in the optimization of technical resources, such as computing,storage, UI (user interface) and/or communication resources, to providee.g. enhanced content, device, service or network usage experience tothe terminal users.

In one other aspect, a method for supplying data to remotely cultivate adigital content reference library, to be executed by an electronicterminal device capable of visually displaying digital content to a userand communicating with a communications network, comprises

-   -   establishing, by automatic observation logic, visual        characteristics data representing the on-screen appearance of        digital content, such as application, shown via a display of the        terminal device, and    -   transmitting the visual characteristics data to a remote server        for updating a digital content reference library thereat linking        content identity information with associated visual        characteristics data in the received or processed form.

Preferably, the method further incorporates determining and sendingcontent identity information identifying the content item, such asapplication or web page, underlying the visual characteristics data tothe remote server, preferably with the data. The determination may becompletely automated or at least partially based on user inputresponsive to e.g. a prompt as mentioned hereinbefore.

In a further aspect, a server arrangement accessible via acommunications network, optionally cellular network and/or the Internet,comprises

a communication module configured to receive content identityinformation and visual characteristics data representing and beingautomatically created based on the on-screen appearance of the content,such as application or other content item, in the terminal device bypredetermined observation logic running in the terminal device forestablishing and transmitting the visual characteristics data,a digital reference library comprising a plurality of entries, an entrylinking content identity information with visual characteristics datarepresenting the on-screen appearance of the content, anda digital reference library management module configured to adapt thedigital reference library by updating an entry therein having regard tosaid content with the visual characteristics data in the received orprocessed form, wherein the entry is addressable with the contentidentity information.

In a further aspect, a terminal device for accessing digital content andcommunicating with a communications network, optionally cellular networkor the Internet, comprises

-   -   a display for visualizing digital content, such as application        running in the terminal, to a user,    -   an observation module configured to automatically establish        visual characteristics data representing the on-screen        appearance of digital content shown via the display, such as        application, and    -   a communication module configured to transmit the visual        characteristics data to a remote server for storage or analysis.

In an embodiment, the observation module is further configured todetermine content identity information, such as application identifier,associated with the digital content based on which the visualcharacteristics data has been established. As alluded to hereinbefore,the identity determination may be automatic or at least partially basedon user input.

In an embodiment, the observation module is configured to capture imagedata such as a screenshot data (screen capture data) of the display viewand establish the visual characteristics data based thereon to representthe on-screen appearance of the content.

In an embodiment, the visual characteristics data includes at least onevector structure or other data structure incorporating a plurality offeatures indicative of the characteristics of the on-screen appearanceof the content at one or more time instants. Preferably, the visualcharacteristics data is or comprises processed, compressed and/orfiltered data established based on image data such as screenshot datarepresenting the on-screen appearance. Advantageously thecharacteristics data is thus provided in a memory-efficient (e.g. lessmemory space consuming) format.

In an embodiment, the observation module is configured to detectswitchover from a previous content, e.g. one application, shown on thedisplay to a next content, e.g. one other application, shown on thedisplay, and utilize such detection as a condition for capturing imagedata to establish the visual characteristics data representing theon-screen appearance of the next content. A programming interface suchas an API (application programming interface) provided by the terminalmay be utilized for detecting the switchover betweencontent/applications. The switchover detection mechanism may, dependingon the technical platform (e.g. iOS™), be capable of detecting a generalswitchover between two different content items such as applications, butis not necessarily able to identify the concerned applications or thenext application, i.e. the identities remain anonymous to the interestedentity such as the observation module.

Preferably, the observation module is further configured to monitorup-to-date overall list, task manager, or other accessible indication ofcontent items, such as applications, that are active or running in theterminal, e.g. active on display or in the background. An interface maybe again provided for the purpose. The content items may be indicatedvia their identity information such as application identifier(application name, for example).

The module is preferably configured to monitor both the aforesaiddetection information and the information about active/running contentindicative of active on-screen and background content to determine acondition and e.g. time period for capturing the image data with properidentity information.

For example, upon detecting a new content item, e.g. application, amongthe active/running ones and detecting a substantially simultaneousswitchover between applications, it may be determined that the newcontent item has just become active in the display (i.e. from thescratch, without already running and being shown in the display prior tothat at some point), whereupon image data captured represents thatparticular content item in the foreground, such as a freshly launchednew application, until the subsequent switchover detected. Theappropriate content identity information associated with the showncontent is determined based on or extracted from the list or otherindication of content items as contemplated hereinbefore.

The previously presented considerations concerning the variousembodiments of the methods may be flexibly applied to the embodiments ofthe arrangement or terminal mutatis mutandis, and vice versa, as beingappreciated by a person skilled in the art.

The utility of the present invention arises from multiple issuesdepending on each particular embodiment thereof. Instead of manuallycreating a data stream of known matches between image (on-screen)characteristics and e.g. corresponding app names, which is error-proneand tedious, basically continuously or regularly executable process, andestablishing a reference library for content matching and related usageanalysis based on that, the terminal devices may be harnessed intoautomatically collecting visual characteristics data based onscreenshots/screen capture data, i.e. image data, and providing the datapreferably in a processed such as compressed, vectorised and/or filteredform to a central entity that associates the identity information of thecontent items with the corresponding visual characteristics data in theentries (or ‘records’) of a maintained collective data structure, or‘library’. The identity information is advantageously automaticallydetermined as well.

The identity information may be determined by the same terminal thatestablishes the visual characteristics data for the particular contenteither completely automatically based on predefined detection logic orin a user-assisted manner, or it may be provided by external entity suchas a server potentially supplying the terminal device with the contentor by an intermediate element such as a proxy.

Solely manual labor-based maintenance of the reference library could bepossible and feasible when the target content space the use of which isto be monitored is very limited, but the present invention enablespractically unlimited scope for different content or media usagemeasurements in terms of a measurable content space of content items asthe user terminals and potential other devices facilitate automaticallyestablishing and keeping the reference library up-to-date by providingvisual characteristics data and identity information relating to thecontent consumed and accessed via the devices also having regard to newcontent items, such as applications, not previously presented in thereference library by any of its existing entries. The mechanism therebyexploits crowdsourcing type of content data acquisition for building andmaintaining the reference library. Manual or ‘expert’ type dataprovision is still naturally possible in addition to the suggestedapproach as explained herein.

By the various embodiments of the present invention, a digital referencelibrary of visual characteristics, or digital fingerprints, may indeedbe conveniently established and also maintained (updated) through theutilization of the automated observation logic at the terminals andoptionally other elements such as network elements providing e.g.identity information. Accordingly, following the necessary initialbuild-up phase of the library and switching over to the maintenance andmatching phase, comparing the contents of the resulting, inherentlycomprehensive reference library, with the inputted visualcharacteristics data to find a match between them for contentidentification purposes becomes more efficient and reliable as theamount of accumulated data in the reference library is both complete andup-to-date.

The present solution thus provides relief in both the creation of newentries in the library for new content (items), such as new apps, andmaintenance of the existing entries/records. Having regard to the actualmatching operations for which purpose the reference library is usuallybeing built in the first place, the comprehensive, dynamically updatedand increasing image or generally visual characteristic data provides inany case incremental value; there's more data and/or more current datato compare the initially anonymous (unidentified) input data with todetermine the actual identity behind the input data for subsequentlyupdating the related content usage statistics, etc.

One major advantageous aspect resides in maximizing or at leastincreasing the variance in the characteristic data hosted by the libraryas different types of terminals (e.g. desktop computers, laptops,tablets, cell phones, phablets, smartphones, etc.) running differentplatforms (e.g. operating system (OS) such as Android™ or iOS™, orWindows™) are typically utilized in the library cultivation andmaintenance procedures. While one very specific, predefinedcharacteristic can provide a good way to map e.g. a home screen (app orOS/platform), it is practically impossible to know beforehand how tovisually trace and thus establish the visual characteristics data forall other possible content the usage of which is worth monitoring. Forexample, some apps can be better matched with color vectors, some appswith data on the screen structure, some apps with shapes or menupositioning etc. Instead of constantly relying on best-effort guesses ofhow to establish the visual characteristics data, a variety ofapproaches may be applied, describing e.g. the top colors (e.g. fivemost used colors), vectors that describe the contrast between differentsides of the screen, vectors that describe the dominant shapes in theimages, and so forth in the automated observation procedures. Therichness of this way versatilely collected data enables scaling up thedata collection platform and overall automation of the reference librarycreation.

The suggested solution also enables clever local identification (e.g.application name, application file name or other identifier) of currenton-screen content on platforms, e.g. at least certain versions of iOS™,not directly providing such identification information to theapplications, such as the observation application, the present inventioncould be embodied as.

Additional benefits of the embodiments of the present invention willbecome clear to a skilled reader based on the detailed descriptionbelow.

The expression “a number of” may herein refer to any positive integerstarting from one (1).

The expression “a plurality of” may refer to any positive integerstarting from two (2), respectively.

Different embodiments of the present invention are disclosed in theattached dependent claims.

BRIEF REVIEW OF THE DRAWINGS

Few embodiments of the present invention are described in more detailhereinafter with reference to the drawings, in which

FIG. 1 illustrates the embodiments of an arrangement and terminal devicein accordance with the present invention in connection with a potentialuse scenario.

FIG. 2 is a block diagram representing the internals of an embodiment ofthe arrangement.

FIG. 3 is a block diagram representing the internals of an embodiment ofthe terminal device.

FIG. 4 is a flow diagram disclosing two embodiments of methods inaccordance with the present invention.

DETAILED DESCRIPTION

FIG. 1 shows, at 100, one merely exemplary use scenario involving anembodiment of an arrangement 114 in accordance with the presentinvention and few embodiments 104 a, 104 b, 104 c of terminal devices inaccordance with the present invention as well.

Network 110 may refer to one or more communication networks such as theInternet, local area networks, wide area networks, cellular networks,etc., which enable terminals 104 a, 104 b, 104 c and server arrangement114 to communicate with each other. Further elements such as a webserver 111 and/or external systems 116 for utilizing data provided bythe arrangement 114 and/or supplying data, optionally control data, tothe arrangement 114, may be included in the established overall system.

The arrangement 114 may be implemented by one or more electronic devicessuch as servers and potential supplementary gear such as a number ofrouters, switches, gateways, and/or other network equipment. In aminimum case, a single device such as a server is capable of executingdifferent embodiments of the method and may thus constitute thearrangement 114 as well. At least part of the devices of the arrangement114 may reside in a cloud computing environment and be dynamicallyallocable therefrom.

The terminals 104 a, 104 b, 104 c may refer to mobile terminals 104 asuch as tablets, phablets, smartphones, cell phones, laptop computers104 b or desktop computers 104 c, for instance, but are not limitedthereto. The users 102 a, 102 b, 102 c may carry mobile devices 104 a,104 b along while heavier or bulkier devices 104 c often remain ratherstatic if are not basically fixedly installed. These devices may supportwired and/or wireless network connections. For example, wired Ethernetor generally LAN (local area network) interface may be provided in somedevices 104 b, 104 c whereas the remaining devices 104 a, 104 b maydominantly support at least cellular or wireless LAN connections.

The terminals 104 a, 104 b, 104 c are provided with observation andcommunication logic e.g. in the form of a computer (processing device)executable software application via a network connection or on aphysical carrier medium such as a memory card or optical disc. Thesoftware may be optionally bundled with other software. The logic isharnessed to obtain image data having regard to digital content shown onthe display device, which is typically integral with the terminal 104 a,104 b, 104 c with reference to e.g. a so-called flat-screen display(LCD, OLED, etc.) or a touchscreen.

The content may refer to, in particular, visual appearance of executablecomputer applications 105 a, 105 b, 106, 108 that are common contentitems running in the terminals 104 a, 104 b, 104 c. Quite typically anapplication has a characteristic visual appearance based on which theapplication itself and/or content shown via it may be recognized forusage tracking purposes, for instance.

Alternatively or additionally, the content may refer to images (or imagefiles), videos (files), or e.g. underlying technical platform such asthe operating system (OS) which may have a characteristic visualappearance (e.g. a so-called desktop view or main view, or some menuview).

Some applications 105 a such as web browsers are occasionally used tovisualize other content such as other applications, i.e. webapplications 105 b. The observation logic may be configured to obtainimage data of the host application 105 a and/or the embedded application105 b.

The obtained image data may be subjected to analysis at the terminals104 a, 104 b, 104 c. For example, a number of characteristic vectors maybe determined therefrom. The analysis results preferably comprise visualcharacteristics data extracted from the image data to be used in thereference library 112 as such and/or in processed form. In someembodiments, the captured image data or portion thereof could be appliedas such in the visual characteristics data.

In addition to visual characteristics data such as data extracted fromthe screen captures or ‘screenshots’, also identity, or‘identification’, information such as application name (e.g. filename,actual application name, and/or other identifier) is preferably obtainedby the terminal 104 a, 104 b, 104 c and linked with the correspondingvisual characteristics data.

The terminals 104 a, 104 b, 104 c are configured to transmit visualcharacteristics data of content items, such as applications, to theserver 114 arrangement optionally in batches. The transmissions may betimed, substantially immediate following the acquisition of the data,and/or be based on other predefined triggers.

Preferably the observation and communication logic acts in thebackground so that any user actions are not necessary for its execution,and the logic may actually be completely transparent to the user (bydefault not visually indicated to the user, for example).

In some embodiments, however, the user may be requested to validate orenter identity information for image data or derived visualcharacteristics data so that it is linked with correct content. Forexample, if content identity information cannot be automaticallydetermined for some technical reason, not at least completely or todesired extent, or the estimated accuracy of the determined identityremains below a predefined level of certainty, the user may be promptedvia the UI of the terminal 104 a, 104 b, 104 c, such as display, toinput or verify the identity information by manual selection or morecomprehensive input (typing in the identifier, for example).

Yet in some embodiments, the external elements 111, 116 may be capableof providing identity information to the arrangement 114. For example,in the case of a web application 105 b supplied by server 111 to theterminal 104 a, 104 b, 104 c, the server 111 may signal the identityinformation of the application 105 b to the arrangement 114 as theserver 111 may be aware of the active on-screen web application 105 brunning in the terminal 104 a. The awareness may be based on e.g. userinteraction with the web application as detected by the server 111.

Reverting to the determination of identity information, such asapplication ID, in the terminals 104 a, 104 b, 104 c, the associatedobservation and communication logic may be configured to request orretrieve such data indicative of the identity of current on-screencontent, when readily available, from a predefined location or viapredefined internal interface, such as API (application programminginterface).

In some embodiments, the identity information relating to the on-screencontent may not be as straightforwardly available to local applications.This may be the case e.g. with certain versions of iOS™ operating systembased platforms.

Even in these scenarios, the terminal 104 a, 104 b, 104 c may beconfigured so as to derive the correct identity information based on aplurality of data inputs one could call as circumstantial evidence.

For example, an indication of a switchover from one application, shownon the display to a next application, shown on the very same display,could be obtainable. The indication would not actually identify theapplications themselves, but just the switchover.

Further, an indication of currently running active visualized orbackground applications (and/or other content, depending on theplatform) could be obtainable e.g. from a task manager.

By combining these two data inputs, i.e. the switchover signal and entryof new application in the list of active/running applications, it couldbe concluded by the observation logic that the new application justadded to the list is very likely the one shown in the display, i.e. aforeground application, until another switchover is detected. Therefore,the visual characteristics data established based on screen view(s)between the instant of first switchover/addition of the new applicationin the list and a subsequent switchover could be automaticallyassociated with the identity of the new application.

In some embodiments, detecting the addition of a new application in thelist of running or active applications could be utilized as identityindication of the on-screen content (i.e. new application) even withoutrelying on the aforesaid switchover or corresponding data. For example,one or few images could be substantially immediately or at least withina relatively short period of e.g. few seconds, captured for establishingthe visual characteristics data to be associated with the identity ofthe new application. This technique could be based on an assumption thata just launched new application typically remains on the screen at leastfor few seconds.

The server arrangement 114 comprises or is at least functionallyconnected to a reference library 112 associating visual characteristicsdata of content items such as applications with related identityinformation, such as application IDs (typically application names). Inpractice, the library 112 may be implemented as one or more databases orother applicable data structures, the actual construction of which isnot in the fulcrum of this invention.

The library 112 comprises, in a functional sense, a plurality of entries113. Each entry, or ‘record’, may contain visual characteristicsdata/number of visual fingerprints such as characteristics (feature)vectors indicative of the on-screen appearance of the particular contentitem. Yet, the entry may contain or indicate (e.g. link) identityinformation of the content such as application name and/or other contentidentifier, such as classification or type data.

During library 112 update or establishment procedure, the serverarrangement 114 preferably obtains both identity information and visualcharacteristics data so that a correct entry associated with theindicated content identity is updated with the visual characteristicsdata.

On the other hand, during a matching procedure, which may take placeupon monitoring usage frequency of different content items fordetermining related statistics and optionally optimizing e.g. marketingefforts based on that, the input may include visual characteristicsdata, i.e. a digital fingerprint, of the content such as application tobe recognized, whereupon the library is searched for a match to retrievethe related identity information. The identity information may be thenutilized in updating e.g. the count figure maintained for the particularcontent item in question.

Further information may be included as well in the library, optionallycomprising e.g. identity/entry-specific frequency or count data havingregard to usage frequency of the content (item) detected and optionallyrelates statistics. Alternatively, such data may be hosted by a separatedata structure 118, which may be addressable with the content identityinformation, for example.

A screen view image of content such as application may in someembodiments be at least partially or coarsely reconstructed utilizingthe visual characteristics data stored.

FIG. 2 is a block diagram representing the internals of an embodiment ofthe arrangement 114 in accordance with the present invention.

The arrangement 114 may be physically established by at least oneelectronic device, such as a server computer. The system 114 may,however, in some embodiments comprise a plurality of at leastfunctionally connected devices such as servers and optional furtherelements, e.g. gateways, proxies, data repositories, firewalls, etc. Atleast some of the included resources such as servers orcomputing/storage capacity providing equipment in general may bedynamically allocable from a cloud computing environment, for instance.

At least one processing unit 202 such as a microprocessor,microcontroller and/or a digital signal processor may be included. Theprocessing unit 202 may be configured to execute instructions embodiedin a form of computer software 203 stored in a memory 204, which mayrefer to one or more memory chips separate or integral with theprocessing unit 202 and/or other elements.

The software 203 may define one or more applications for maintaining thereference library by inserting data thereto or fetching data therefromand optionally for determining required usage statistics such as usefrequencies or counts of various content items. A computer programproduct comprising the appropriate software code means may be provided.It may be embodied in a non-transitory carrier medium such as a memorycard, an optical disc or a USB (Universal Serial Bus) stick, forexample. The program could be transferred as a signal or combination ofsignals wiredly or wirelessly from a transmitting element to a receivingelement.

One or more data repositories such as database(s) 112, 118 of preferredstructure may be established in the memory 204 for utilization by theprocessing unit 202. The repositories may physically incorporate e.g.RAM memory, ROM, Flash or other non-volatile (RAM) memory, magnetic/harddisc, memory card, etc. The repositories host the reference library, forexample.

The UI (user interface) 206 may provide the necessary control and accesstools for controlling the arrangement (e.g. definition of librarymanagement rules or data analysis logic) and/or accessing the datagathered and calculated (analysis results, statistics, identityinformation, visual characteristics data, etc.). The UI 206 may includelocal components for data input (e.g. keyboard, touchscreen, mouse,voice input) and output (display, audio output) and/or remote input andoutput optionally via a web interface, preferably web browser interface.The system may thus host or be at least functionally connected to a webserver, for instance.

Accordingly, the depicted communication interface(s) 210 refer to one ormore data interfaces such as wired network (e.g. Ethernet) and/orwireless network (e.g. wireless LAN (WLAN) or cellular) interfaces forinterfacing a number of external devices and systems with the system ofthe present invention for data input and output purposes, potentiallyincluding control. The arrangement 114 may be connected to the Internetfor globally enabling easy and widespread communication therewith. It isstraightforward to contemplate by a skilled person that when anembodiment of the arrangement 114 comprises a plurality of functionallyconnected devices, any such device may contain a processing unit,memory, and e.g. communication interface of its own (for mutual and/orexternal communication).

Primarily considered from a functional or conceptual standpoint, see thelower block diagram at 215, the arrangement 114 comprises a librarymanagement module 212 for managing the library 112 and keeping itup-to-date based on the received identification and visualcharacteristics, or ‘fingerprint’, data. Yet, it may process thereceived data into storage format of the library 112 by compressing it,extracting features therefrom, etc. The module 212 is preferably alsoconfigured to retrieve the requested data from the library 112 foranalysis purposes, such as identity information corresponding to anindicated visual characteristics data. For the purpose, the module 212may apply a predefined matching algorithm and related decision criteriafor determining the identity information (if any) for the indicatedvisual characteristics data input.

The reference library 112 is the element or entity that stores thevisual characteristics data and the related identity information, or atleast indication such as link thereto. It 112 thus associates detectedvisual characteristics of content items with corresponding identityinformation of the items. It 112 may host further data as well such asusage frequency or ‘count’ data that may be utilized in determiningvarious usage statistics. In terms of data management and storage, thelibrary 112 may be physically constructed from one or more datastructures such as databases, optionally physically distributedstructures or databases, which are then functionally linked.

Communication module 216 responds to external data inquiries (e.g.requests of desired statistics, identity information and/or visualcharacteristics data) and receives external data such as visualcharacteristics data from the terminals and associated identityinformation. The interface(s) 210 are utilized for these purposes.

Finally, a reporting or analysis module 218 may be included (marked withbroken line) for executing analysis actions on the stored data 112 suchas determination of desired content or media usage statistics and/ordigital audience estimates (user estimates).

FIG. 3 is a block diagram representing the internals of an embodiment ofthe terminal device 104, which may refer to any terminal of 104 a, 104b, 104 c, for example. In congruence with the server 114, the terminal104 includes a processing unit 302 and memory 304 for executing andstoring program instructions 303 for gathering and providing visualcharacteristics data and preferably identity information, respectively.Yet, the memory 304 stores other data such as the visual characteristicsdata. Communication interface 310 may include at least one wired (e.g.Ethernet or other LAN) and/or wireless (e.g. WLAN, cellular such as 3Gor 4G) communication interface, typically a transceiver for transferringdata, including receiving content and transmitting visualcharacteristics data and preferably associated identity information.

The UI 306 refers to at least a display (screen), the on-screen views ofwhich are used to determine visual characteristics data to be associatedwith identity information of the content items producing the views.Still, the UI 306 may include e.g. user input interface such astouchscreen functionality in connection with the display, a separatekeypad, keyboard, mouse, voice input, etc. Such input interface may beconfigured to obtain e.g. definition or validation/verification ofcontent identity from the user.

Again, at the lower portion 315 of the figure, more functional orconceptual approach is taken. An observation module 312 is configured toobtain the on-screen display data, i.e. image data, e.g. via applicableinternal interface(s) of the terminal 104 in order to establish thevisual characteristics data. Predefined feature or feature vectorcriteria may be applied to establish the data to be transmitted from theoriginal on-screen image data, for example.

To monitor on-screen activity or view in the first place, the deviceshall include a display 314, which may refer to a conventional displaysuch as LCD (liquid-crystal display) or OLED (organic LED) display, or afunctionally corresponding data visualization equipment, e.g. aprojector, so that the visual characteristics data is really generatedand thus obtainable via the interface(s) or through other routesinternally at the device 104.

The visual characteristics data may in some embodiments include a numberof selected screen captures and/or data derived therefrom. Further, theobservation module 312 is preferably configured to determine at leastpart of the identity information for the visual characteristics data,such as application name of the on-screen application in question.

The communication module 316 transmits the visual characteristics dataand preferably identity information to the arrangement 114 via availablephysical communication interface(s) 310. It may further receive controldata and/or update data for the observation logic from external sourcessuch as the arrangement 114.

The analysis module 318 may be included to execute various optionalmonitoring and/or analysis activities in the terminal 104. For example,data traffic, application data, voice call data, messaging data, and/ore.g. network identities in range may be monitored, analyzed and/orlogged. Such data may also be transmitted to external entities such asarrangement 114 for further analysis or storage.

Having regard to different embodiments of the modules of FIGS. 2 and 3,a person skilled in the art will appreciate the fact that the abovemodules and associated functionalities may be realized in a number ofways. A module may be divided to functionally even smaller units or twoor more modules may be integrated to establish a larger functionalentity. In case the arrangement 114 comprises several at leastfunctionally connected devices, the modules may be executed by dedicatedone or more devices or the execution may be shared, even with dynamicallocation, among multiple devices e.g. in a cloud computingenvironment.

FIG. 4 is a flow diagram 400 disclosing two embodiments of methods inaccordance with the present invention.

The top half of the figure refers to an embodiment of a methodincorporating activities typically executed by terminal(s) andoptionally in some parts, other elements such as network elements (e.g.a web server or application server), whereas the bottom part refers toan embodiment of a method incorporating activities executed by theserver arrangement.

At 404, referring to a start-up phase, the necessary preparatory actionsare executed, which refers to both terminal and server side embodiments.The system hardware, such as at least one server apparatus withsufficient data processing, storage and communication capabilities, maybe acquired and set up by loading it with appropriate control software.The communication connections relative to external systems may beestablished and tested. Run-time parameters and e.g. transaction andidentifier analysis logic may be determined. A number of, typically aplurality of, terminals may be provided with compatible observationlogic in the form of stand-alone or bundled software, for example. Theassociated configuration may be adjusted, optionally by the user.

Item 406 is about establishing at each concerned terminal, by automatic,preferably autonomously functioning observation logic, visualcharacteristics data representing the on-screen appearance of digitalcontent, such as application, shown via a display of the terminaldevice.

At 410, the content identity information is determined by theobservation logic running in the terminal and/or by an external entitysuch as application data-providing server, which may refer to a webserver, for instance.

Item 408 is about transmitting the visual characteristics data andidentity information, preferably by the same terminal, to a remoteserver for updating a digital content reference library thereat linkingcontent identity information with associated visual characteristics datain the received or processed form.

At 412, content identity information and visual characteristics data arereceived at the server.

Item 414 is about adapting the digital reference library by updating anentry therein having regard to said content with the visualcharacteristics data in the received or processed form, wherein theentry is preferably addressable with the content identity information.

It may be determined whether the reference library already includes anentry for the particular content item in question based on the receivedidentity information, and in the case of previously non-existing contentitem establishing and in the case of already-existing content itemmodifying or supplementing an entry in the reference library for theindicated application/content, which may refer to adding new visualcharacteristics data, changing the existing data, or replacing theexisting data with the new data, for example.

Items 416-422 relate particularly to the utilization of the establisheddigital content fingerprint library in the monitoring and keeping trackof content usage at the terminals when the input data does not containcontent identity information.

At 422, the server receives visual characteristic data, such as imagecharacteristic vector(s), of the on-screen content from a terminaldevice comprising monitoring logic for capturing/determining andforwarding such data.

At 416, a search is executed in the library to find matching data and inthe case of a match to retrieve the related identity information, whichis then done at 420.

Related usage data deliverables such as log(s) may be updated includinge.g. usage count/frequency data and outputted to local or externalrecipients at 418, which may refer to different applications, databases,computers, computer systems, digital services, etc.

The statistics may be utilized in determining e.g. digital marketingefforts or optimizing technical resources (computation, UI, storage,communication, etc.) for the systems serving the users/terminals.

Item 418 may be executed also in scenarios where the server is directlyprovided with content identity information together with the visualcharacteristics data, with reference to items 412-414, for example.Maintenance of content usage such as application usage frequency/countfigures or other related statistics may begin; usage log may be updatedwith e.g. application identifier and related time data, etc.

At 424, the method execution is ended.

The dotted, only exemplary, loop-back arrows reflect the likelyrepetitive nature of various method items when executed in differentreal-life and potentially also substantially real-time scenarios whereinvarious on-screen content is consumed every now and then, whereupon theestablished system of terminal(s) and server(s) may perform relatedobservation, communication and analysis tasks in response.

The scope is defined by the attached independent claims with appropriatenational extensions thereof having regard to the applicability of thedoctrine of equivalents, etc.

1. A method for cultivating a digital reference library for digital content, such as a plurality of applications, linking content identity information and visual characteristics data, comprising receiving content identity information and visual characteristics data representing and being automatically created based on the on-screen appearance of the content, such as application or other content item, in the terminal device by predetermined observation logic running in the terminal device for establishing and transmitting at least visual characteristics data, and adapting the digital reference library by updating an entry therein having regard to said content with the visual characteristics data in the received or processed form, wherein the entry is addressable with the content identity information.
 2. The method of claim 1, wherein content identity information and visual characteristics data associated with the content are automatically determined by the observation logic running in terminal device.
 3. The method of claim 1, wherein the content identity information is at least partially created based on explicit user input in the terminal device.
 4. The method of claim 1, wherein the content identity information is received from a network element, optionally a web or application server.
 5. The method of claim 1, comprising determining whether the reference library includes an entry for the particular content item in question based on the received identity information, and in the case of previously non-existing content establishing, and in the case of already-existing content supplementing, an entry in the reference library for the indicated application.
 6. The method of claim 1, wherein the content identity information indicates at least one element selected from the group consisting of: application identifier of the on-screen application, application name, application manufacturer, application type or class, web page address or name, web site address or name, event occurred, activity performed, operating system, and payment or purchase action.
 7. The method of claim 1, wherein the content item the identity information and visual characteristics data of which are provided in the library refers to at least one element selected from the group consisting of: application, native application, web application, browser, web site, web page, operating system, image and video.
 8. The method of claim 1, comprising receiving visual characteristics data representing the displayed on-screen appearance of content item in a terminal device without application identity information, and matching the data is matched as is or in processed form with the existing data in the entries of the library to find the identity of a matching content item.
 9. The method of claim 8, further comprising determining usage statistics based on the identified, visual characteristics data.
 10. A method for supplying data to remotely cultivate a digital content reference library, to be executed by an electronic terminal device capable of visually displaying digital content to a user and communicating with a communications network, comprising establishing, by automatic observation logic, visual characteristics data representing the on-screen appearance of digital content, such as application, shown via a display of the terminal device, and transmitting the visual characteristics data to a remote server for updating a digital content reference library thereat linking content identity information with associated visual characteristics data in the received or processed form.
 11. The method of claim 10, further comprising determining and sending, preferably automatically, content identity information identifying the content item underlying the visual characteristics data to the remote server.
 12. The method of claim 11, wherein the determination is at at least partially based on user input optionally responsive to a prompt indicated audibly and/or shown on the display.
 13. A server arrangement accessible via a communications network, optionally cellular network or the Internet, comprising a communication module configured to receive content identity information and visual characteristics data representing and being automatically created based on the on-screen appearance of the content, such as application or other content item, in the terminal device by predetermined observation logic running in the terminal device for establishing and transmitting the visual characteristics data, a digital reference library comprising a plurality of entries, an entry linking content identity information with visual characteristics data representing the on-screen appearance of the content, and digital reference library management module configured to adapt the digital reference library by updating an entry therein having regard to said content with the visual characteristics data in the received or processed form, wherein the entry is addressable with the content identity information.
 14. The arrangement of claim 13, wherein the management module is further configured to execute a matching algorithm among the data entries in the library for finding identity information for visual characteristics data received without content identity information.
 15. The arrangement of claim 13, further comprising an analysis module configured to determine a number of content usage statistics, optionally frequency or count data, based on the visual characteristics data received.
 16. A terminal device for accessing digital content and communicating with a communications network, optionally cellular network or the Internet, comprising a display for visualizing digital content, such as application running in the terminal, to a user, an observation module configured to automatically establish visual characteristics data representing the on-screen appearance of digital content shown via the display, such as of application, and a communication module configured to transmit the visual characteristics data to a remote server for storage or analysis.
 17. The device of claim 16, wherein the observation module is further configured to determine, preferably automatically, content identity information, such as application identifier, associated with the digital content item based on which the visual characteristics data has been established.
 18. The device of claim 16, wherein the observation module is configured to capture image data such as screen capture data of the display view and establish the visual characteristics data based thereon to represent the characteristic on-screen appearance of the content in a memory-efficient manner.
 19. The device of claim 16, wherein the visual characteristics data includes at least one vector structure or other data structure incorporating a plurality of features indicative of the characteristics of the on-screen appearance of the content at one or more time instants.
 20. The device of claim 16, configured to detect a new content item, optionally application, among the executed ones, and a switchover between applications, and subsequent to detection of the two, configured to capture image data shown on the display and associate with the identity information of the new content item.
 21. A computer program comprising code means adapted, when executed on a computer, to perform the method of claim
 1. 22. A carrier medium comprising the computer program of claim
 21. 